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WINDOWS API TRAPPING SYSTEM 

Cross-Reference to Related Applications 

This application is based on U.S. provisional patent application No. 60/028,339, 
filed on October 1 1, 1996. 

Background of The Invention 

1 . Field of the Invention 

This application relates to the field of software and more particularly to the field of 
managing aspects of an interface between software and the underlying operating system. 

2. Description of Related Art 

Many conventional operating systems provide a formalized Application 
Programming Interface (API) that allows application programmers to make calls to 
software routines that perform a variety of system-wide functions. Using the API 
fecilitates writing application programs by decreasing the amount of code that application 
programmers need to provide and, at the same time, providing standardization of routines 
that are used by many of the applications. 

However, in some instances, it is necessary to modify an API call in order to, for 
example, perform specialized functions that are not provided by the operating system or to 
keep track of certain types of API calls. For some operating systems, it is not difficult to 
intercept and monitor API calls. For example, under MS-DOS, most APIs use interrupts 
and are thus easy to intercept. In other instances, intercepting and trapping API calls can 
be challenging for an application programmer. For example, in some Microsoft Windows 
Environments (Winl6 and Win32), the APIs use exported functions which are connected 
to the application at application load time via static or dynamic linking. This linkage 
process is done by the internal OS routines and undocumented data structures which are 
usually inaccessible to outside (non-Microsoft) software developers. In addition, newer 
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versions of the Windows OS actively thwart API intercepting methods described in the 
programming literature for the earlier versions of the OS. Li some instances, a 
technological race is unfolding between the operating system developers and the third 
party (application) developers who need to provide OS/applications enhancements 
unfbrseen (or undesired) by the OS developers. 



A method for intercepting Windows 3.x APIs based on patching the entry point of 
the API fbnction with a JMP instruction is known. In this system, the interceptor first 
obtains the address of the target API function via a call to GetProcAddressO. Note that 
this step has been actively thwarted by the OS in Win95 for many key API functions, 
although within months the workarounds for much of the thwarting attempts have been 
published. Following getting the procedure address, the interceptor removes the 
write-protection of the obtained memory address (which is a code address, thus it is set by 
the OS as read/execute-only). Following this, the interceptor patches the API entry point 
with JMP Intercepts™ instruction, where Intercepts™ is a service function in the 
interceptor's code that monitors and/or processes the API calls. 

After the interceptor has set up the API call in the manner described above, an 
application or OS calls the API entry point. The JMP Intercepts™ instruction at the 
entry point address transfers control to the InterceptSrvO function and the InterceptSrvO 
processes the call (accessing as necessary the function arguments on the stack) and, upon 
completion, either returns control to the caller or passes control to the original API 
function. Returning control to the caller is straightforward. Passing control to the API, 
on the other hand, requires that the InterceptSrvO remove the patched in JMP 
Intercepts™ from the API entry point and restore the original opcode bytes (the original 
five bytes that were present in the API before the patch was inserted). Then, the 
InterceptSrvO pushes all original arguments from trie-caller's stack on to the current stack 
(interceptor's stack). Finally, the InterceptSrvO calk the original API function. When the 
original API returns, the InterceptSrvO saves the return value into a local variable, the 
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InterceptSrvO reinserts the JMP Intercepts™ patch at the API entry point, and the 
InterceptSrvO returns the saved return value to the original caller. 

The technique described above is used in many commercial application enhancers. 
However, it has many drawbacks. For one thing, the division oflabor between the set up 
portion and the run time portion is highly inefficient since the set up portion is executed 
only once (at the interceptor's load time) while the run time portion is executed many 
times (from the load time onward). In a more efficient system, as much of the work as 
possible would be shifted from the run time portion to the set up portion. I* addition, for 
a processor running MS Windows on an Intel processor, the necessary replacement of the 
JMP Intercept opcodes requires a write operation of five bytes, which can not be done in 
a single processor instruction. Hence, the replacement leaves a short interval of instability 
between the two write instructions, during which the API entry point has invalid 
instructions and a hardware interrupt at that moment could initiate reentry into the API, 
which will probably crash the system. 

On the other hand, disabling the interrupts in Windows application mode (CPU 
ring 3 code where these actions are occurring) is a highly expensive operation due to 
system control over the CPU interrupt flag, which triggers an elaborate exception process 
On CPU ring 0). The complex ring transition and the exception process, which may take 
as much or more time than all the rest of processing above, would occur twice. Therefore 
most commercial interceptors (as well as the published code) choose the tradeoff with the 
bstability allowed, in order not to pay the disproportionate performance cost associated 
with disabling interrupts. 

Another disadvantage of the technique described above is that, during certain time 
intervals, another call to the monitored API will be missed by the interceptor. Since many 
Windows APIs perform checks on the task queue within and can (and often do) switch to 
another thread/task, the missing of intercepts is a real problem for interceptors which 
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require processing on every call to the API (especially those implementing security 
features). Also, the overwriting of the API entry point on every call is unsafe when 
multiple interceptors exist on the same system. For example, while the system is 
processing the original API call, a switch to another task or thread can (and often will) 
5 occur. If the second task inserts its own intercept for the same API (or spawns a program 
which does that), then return to the first interceptor will destroy the new intercept, thus 
permanently disabling operation of the new intercept Additionally, if the first intercept 
unloads, followed by an unload of second intercept, then the API entry will be left pointing 
to the non-existent first intercept and the system will crash when the API is invoiced. 

10 Another disadvantage of the technique described above is that some Windows API 

fiinctions (e.g. memory allocation and protection) are sensitive to the source of the call, 
Le, the Windows API code will check where the call is made from and, based on 
knowledge of Microsoft's sources of calls, will work differently if called from third party 
applications as opposed to particular Microsoft sources. This behavior is, among other 
reasons, related to the active thwarting of the third party interceptors mentioned above. 
Since the intercept technique described above changes the original source of the API call 
and makes the interceptor appear to Windows as the source of the API call, the Windows 
API processing may operate differently, often malfunctioning in a way that leads to 
instabilities and system crashes. This makes the technique described above unsuitable for 
20 intercepting some of the Windows APIs. 



15 



25 



Summary Of The Invpntinn 

According to the present invention, supplementing a software routine loaded in 
computer memory includes loading an additional routine into the computer memory, 
providing relocated opcodes by relocating a number of bytes from a relocatable portion of 
the software routine to an other memory location where the number of bytes corresponds 
to an integral number of instructions of the relocatable portion, causing program control 
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to flow from the additional routine to the relocated opcodes, causing program control to 
flow from the relocated opcodes to a memory address immediately following the 
relocatable portion, and causing program control to flow from the relocatable portion to 
the additional routine. 

5 Causing program control to flow from the additional routine to the relocated 

opcodes may include placing the relocated opcodes at a location in the computer memory 
that immediately follows the additional routine. Causing program control to flow from the 
relocated opcodes to the memory address immediately following the relocatable portion 
may include placing a program control instruction at a location in the computer memory 

10 immediately following the relocated opcodes. The program control instruction may be an 
unconditional jump instruction. Causing program control to flow from the relocatable 
portion to the additional routine may include placing a program control instruction at a 
memory location corresponding to a source address of the relocatable portion of the 
software routine. The program control instruction may be an unconditional jump 

15 instruction. Providing the relocated opcodes may include relocating a number of bytes 

that is at least equal to an amount of bytes required for the unconditional jump instruction. 

Following providing the relocated opcodes, it is possible to resolve any opcodes 
contained therein that reference relative displacements between the relocated opcodes and 
opcodes contained in the software routine. The software routine and additional routine 
20 may be API's that run under the Microsoft Windows operating system. The additional 
routine may be configured to load at a predetermined address in the computer memory. 

According further to the present invention, supplementing a Windows API loaded 
in computer memory includes loading an additional routine into the computer memory, 
providing relocated opcodes by relocating a number of bytes from a relocatable portion of 
25 the API to an other memory location where the number of bytes corresponds to an 

integral number of opcodes of the relocatable portion, causing program control to flow 
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from the additional routine to the relocated opcodes, causing program control to flow 
fiom the relocated opcodes to a memory address immediately following the relocatable 
portion, and causing program control to flow from the relocatable portion to the 
additional routine. According further to the present invention, a software program that 
supplements a Windows API loaded in computer memory includes an additional routine 
that is loaded into the computer memory, first means for providing relocated opcodes by 
relocating a number of bytes from a relocatable portion of the API to an other memory 
location where the number of bytes corresponds to an integral number of instructions of 
the relocatable portion, second means, coupled to the first means and to the additional 
routine, for causing program control to flow from the additional routine to the relocated 
opcodes, third means, coupled to first means and to the API. for causing program control 
to flow from the relocated opcodes to a memory address immediately following the 
relocatable portion, and fifth means, coupled to the API and to the additional routine, for 
causing program control to flow from the relocatable portion to the additional routine. 

In the intercept install phase, the technique described herein disassembles the target 
API entry code and relocates (based on the semantics of the instructions found there) the 
whole instructions from the API entry into the interceptor's memory. Then, in the intercept 
operation phase, when control needs to be passed to the original API function, instead of 
having to swap back and forth the overlayed API entry opcodes, the intercept simply 
passes control to the second (relocated) copy of the API entry code, which, upon 
completion, passes control to the next section of the original API code (the section which 
follows the relocated section). 

This method thus shifts the division of labor heavily toward the install phase of the 
intercept, relieving the active phase of the intercept by eUminating many of the steps 
described in connection with the prior art system and the performance, safety and system 
stability drawbacks associated with them 
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The technique described herein has many advantages over conventional trapping 
systems. One advantage is that all trapping work, except for a minimum amount of work 
necessary to transfer control to the interceptor and back, is done only once at install time, 
relieving the performance burden from the run time activity of the interceptor. In addition, 
5 no opcode swapping is done during the existence and activation of the traps. This 

eliminates performance, stability and safety drawbacks resulting from the activity found in 
some conventional systems. Furthermore, since the execution of the some of the trapping 
code occurs on the stack of the original caller, there is no need to copy API function 
arguments to the interceptor's stack. By executing much of the code on the original 
10 caller's stack, the source of the call will appear to the Windows API as if it came from the 
original caller, therefore resolving problems associated with Windows behaving differently 
depending on the identity of the caller. By not copying API entry opcodes back and forth 
at each API call, no window of system instability is created. Instead, time-consuming 
precautions (eg., disabling interrupts) occurs once at intercept install time. 

15 There are additional advantages. Since the trap opcodes are never removed during 

processing of the original API, the possibility of missing API calls b significantly 
decreased, as described above. Therefore, the system described herein is an excellent 
choice for situations where the interceptor must see all of the API calls to the target API 
function to operate properly or reliably. The problem of multiple interceptors setting traps 
20 while processing of the trap is going on is resolved, since the new interceptor will always 
see fixed opcodes at the API entry point, thus the second interceptor will not be forcibly 
disabled, as in conventional techniques. Also, since the setting and removal of the traps 
occur only once at load/unload time of interceptor, the problem of dangling intercept (with 
target of JMP InterceptSrv already unloaded) can be avoided since the interceptor can 
afford more detailed, time consuming, checks for safe removal of the intercepts (e.g. by 
refusing to unload itself if it is not the last interceptor). Doing these checks in the old 
trapping system is not only time consuming on every API call, but it is in most cases 
extremely difficult, if not inherently impossible, since it would require that the interceptor 
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refiise execution of the original API call, which will cause malfunction in the calling 
application. In addition, the system described herein protects against unauthorized 
canceling of installed API security functions since canceling a new API task installed using 
the technique described herein, without restoring the original opcodes of the original API, 
5 will likely cause the system to crash. 

Brief Descrintion Of Drawing* 

FIG. 1 is a diagram illustrating a relationship between an old API and a new API 
according to the present invention. 

FIG. 2 is a flow chart showing steps that are performed to install the new API 
10 according to the present invention. 

FIG. 3 is a flow chart illustrating steps that are performed to remove the new API 
that is installed using the steps of FIG. 2. 

Detailed Descrintion of the-. Preferred Ktnhnrfi^ ^nTfs) 

Refer to FIG. 1, a diagram 10 illustrates a relationship between an old API 12 and 

15 a new API 14. The old API 12 represents an existing API provided with an operating 

system, such as MS Windows 95. The new API 14 represents an API that is provided for 
use in connection with, for example, an applications program. As discussed in more detail 
below, the new API 14 can be executed instead of the old API 12 or can be executed in 
addition to the old API 12. Note that, in some instances, the new API 14 and/or the old 

20 API 12 may be referred to herein as a "routine". However, the term "routine" should not 
be understood as referring to a single, unitary, block of code but, instead, should be 
understood to refer to a collection of code that may be provided in a plurality of blocks 
that may make calls or jumps therebetween. Note also that the specific functionality 
provided by the new API 14 is a design choice but may, in some instances, including 

25 saving and restoring registers used by the caller and/or the old API 12. 
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In order to execute the new API 14, the old API 12 is patched with an 
unconditional jump instruction 16 that transfers control from the old API 12 to the 
beginning of the new API 14. If the new API 14 is executed instead of the old API 12 
(Le., the old API 12 is not to be executed), then a return to the calling routine occurs at 
the end of the new API 14, as indicated by the dotted line shown at the end of the new 
API 14. Note that, as will be apparent to one of ordinary skill in the art, other suitably 
equivalent control flow instructions may be used in place of the unconditional jump 
instruction 16. 



If the old API 12 will be executed inaddition to the new API 14, then opcodes 13 
that were located at a relocatable portion of the old API 12 Qn this case the beginning of 
the old API 12) become relocated code 1 8 that is placed at the end of the new API 14. As 
discussed in more detail hereinafter, any relative offsets between opcodes within the 
relocated code 18 and opcodes in the remainder of the old API 12 are adjusted, as 
appropriate. Note that, as will be apparent to one of ordinary skill in the art, it is possible 
to place the relocated code 18 at an other portion of memory and then use an appropriate 
control flow instruction at the end of the new API 14 to transfer program control from the 
new API 14 to the relocated code 18. 

Immediately following the end of the relocated code 18 is an unconditional jump 
instruction 19 that transfers program control to the portion of the old API 12 immediately 
following the relocatable portion of the old API 12, marked on the diagram 10 with the 
address "CONT". Thus, if both the old API 12 and the new API 14 are to be executed, 
then the calling routine calls the old API 12 which jumps, via the jump instruction 16, to 
the beginning of the new API 14 which then executes and, at the end thereof, executes the 
opcodes of the relocated code 18 followed by the jump instruction 19 that jumps back to 
the remainder of the old API 12. Note that the relocated code 18 and the portion of the 
old API 12 beginning at the CONT address constitute the entirety of the old API 12. 
Also note that, as will be apparent to one of ordinary skill in the art, other suitably 
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equivalent control flow instructions may be used in place of the unconditional jump 
instruction 19. 

Referring to Fig. 2, a flow chart 20 illustrates steps for making patches that cause 
execution of the new API 14 when an application program or the operating system calls 
the old API 12. Processing begins at a first step 22, where a byte from the beginning of 
the old API 12 is fetched. Following the step 22 is a test step 24 which determines if a 
whole instruction (as opposed to a partial instruction) has been fetched. This 
determination is made in a conventional fashion by, for example, disassembling the retched 
bytes. Note that it is necessary to fetch an integral number of instructions from the 
relocatable portion of the old API 12 since it is not possible to execute a partial 
instruction. 

If it is determined at the test step 24 that one or more whole instructions have not 
been fetched, then control passes from the test step 24 back to the step 22 to fetch another 
byte. Otherwise, if an integral number of instructions have been fetched, then control 
passes from the test step 24 to a test step 26 which determines if enough bytes have been 
fetched from the relocatable portion of the old API 12 to accommodate the unconditional 
jump instruction 16 that transfers control from the old API 12 to the new API 14. In 
some embodiments, the required number of bytes is five. However, the test step 24 
preceding the test step 26 makes it possible that a number of bytes greater than five will 
have been fetched since it is necessary that a number of bytes corresponding to an integral 
number of instructions be fetched from the relocatable portion of the old API 12. 

If it is determined at the test step 26 that enough bytes have not been fetched, then 
control passes from the test step 26 back to the step 22 where another byte is fetched. 
Otherwise, if enough bytes have been fetched, then control passes from the test step 26 to 
a step 28. Note that it is not possible to reach the test step 26 without having retched a 
number of bytes corresponding to an integral number of instructions. This is because it is 
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not possible to execute the test step 26 without having passed the test at the step 24, 
which determines that the number of fetched bytes corresponds to a whole number of 
instructions. 



At the step 28, the bytes that have been fetched are moved from the relocatable 
5 portion of the old API 12 to the end of the new API 14 and any opcodes in the relocated 
code 18 that refer to relative offsets are resolved. Note that opcodes in the relocated code 
18 that contain a relative offset, such as a jump relative or a call relative, may need to be 
modified when the opcodes are relocated. Also note that, the relative positions within 
memory of the old API 12 and the new API 14 should not change after the API's 12, 14 
10 are loaded in memory. 

Following the step 28 is a step 30 where the unconditional jump instruction 19 is 
added to the end of the new API 14. As discussed above, the unconditional jump 
instruction 19 causes control to return back to the portion of the old API 12 that follows 
the relocatable portion of the old API 12. Following the step 30 is a step 32 where the 
15 unconditional jump instruction 16 is added to the beginning of the old API 12 so that 
when an application program or the operating system calls the old API 12, the 
unconditional jump instruction 16 from the old API 12 to the new API 14 will be 
executed. 

As discussed above, it is possible that the new API 14 entirely replaces the old API 
20 12 so that no part of the old API 12 needs to be executed once the new API 14 has been 
provided. In that case, an alternative patch is provided. As shown in FIG. 2, control 
passes from the test step 26 to a step 34 where a return instruction is added to the end of 
the new API 14 (if a return instruction is not already found at the end thereof). Following 
the step 34, control passes to the step 32 where the unconditional jump instruction 16 is 
25 added to the relocatable portion of the old API so that a call to the old API 12 wDl cause 
program control to flow from the old API 12 to the new API 14. 
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Note that, in some instances, it may be unadvisable to relocate the code that is at 
the beginning of the old API 12. For example, there may be other instructions within the 
old API 12 that reference the code located at the beginning thereof In those cases, it is 
possible to use other portions of the old API 12 as the relocatable portion. For example; it 
would be possible to relocate bytes following the first N microprocessor instructions, in 
the manner described above, and replace the relocated bytes with the unconditional jump 
instruction 16. 

The code that executes the patching step illustrated by the flow chart 20 may be 
written in a conventional computer source language, such as C++, and compiled in a 
conventional manner similar to compilation of other Microsoft Windows DLL's. In some 
instances, the preferred base address of the new API 14 may be set to a value that will 
cause the new API 14 to always load at the same address for all the processes which use 
the new API 14. Thus, the new API 14 may be shared so that the new API 14 is loaded in 
memory only once, even when used by multiple processes. 

Refer to FIG. 3, a flow chart 40 illustrates steps that are performed when a process 
that uses the new API 14 is removed from memory. Note that, under the Windows 
environment, a special routine (MS Main) is called when a process is removed from 
memory. The MS Main routine provides the application with an opportunity to do 
cleanup including, in this instance, restoring the old API 12. 

Processing begins at a first test step 42 where it is determined if the process being 
removed is the last process that uses the new API 14. If not, then the old API 12 and the 
new API 14 are not modified and no cleanup is done, since the new API 14 must remain 
to be used by the other processes. Otherwise, control passes from the test step 42 to a 
step 44 where the relative instructions of the relocated code 18 are modified bade to the 
original state prior to restoring the relocated code into the old API 12. Following the step 
44 is a step 46 where the relocated code 18 is restored into the old API 12, thus 
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oveiriding the unconditional jump instruction 16 that was provided to the old API 12 
when the old API 12 was patched. Once the relocated code 18 b restored to the old API 
12, then a call to the old API 12 will not result in execution of the new API 14. 

Note that, although the invention has been illustrated herein using APIs with the 
Windows operating system, it would be straight-forward for one of ordinary skill in the art 
to adapt the system described herein to other operating systems and other types of 
routines. 

While the invention has been disclosed in connection with the preferred 
embodiments shown and described in detail, various modifications and improvements 
thereon will become readily apparent to those skilled in the art Accordingly, the spirit 
and scope of the present invention is to be limited only by the following claims. 
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Claimfe) 

1. A method of supplementing a software routine loaded in computer memory, 
comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the software routine to an other memory location, 
the number of bytes corresponding to an integral number of instructions of 
the relocatable portion; 

(c) causing program control to flow from the additional routine to the 
relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

2. A method, according to claim 1, wherein causing program control to flow from the 
additional routine to the relocated opcodes includes placing the relocated opcodes at a 
location in the computer memory that immediately follows the additional routine. 

3. A method, according to claim 1, wherein causing program control to flow from the 
relocated opcodes to the memory address immediately following the relocatable portion 
includes placing a program control instruction at a location in the computer memory 
immediately following the relocated opcodes. 

4. A method, according to claim 3, wherein the program control instruction is an 
unconditional jump instruction. 

5. A method, according to claim 1, wherein causing program control to flow from the 
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relocatable portion to the additional routine includes placing a program control instruction 
at a memory location corresponding to a source address of the relocatable portion of the 
software routine. 

6. A method, according to claim 5, wherein the program control instruction is an 
unconditional jump instruction. 

7. A method, according to claim 6, wherein providing the relocated opcodes includes 
relocating a number of bytes that is at least equal to an amount of bytes required for the 
unconditional jump instruction. 

8. A method, according to claim 1, further comprising: 

(0 following providing the relocated opcodes, resolving any opcodes 
contained therein that reference relative displacements between the 
relocated opcodes and opcodes contained in the software routine. 

9. A method, according to claim 1, wherein the software routine and additional routine are 
API's that run under the Microsoft Windows operating system. 

10. A method, according to claim 9, wherein the additional routine is configured to load at 
a predetermined address in the computer memory. 

1 1. A method of supplementing a Windows API loaded in computer memory, comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the API to an other memory location, the number of 
bytes corresponding to an integral number of instructions of the relocatable 
portion; 

(c) causing program control to flow from the additional routine to the 
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relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

12. A software program that supplements a Windows API loaded in computer memory, 
comprising: 

an additional routine that is loaded into the computer memory; 

first means for providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the API to an other memory location, the number of bytes 
corresponding to an integral number of instructions of the relocatable portion; 

second means, coupled to the first means and to the additional routine, for causing 
program control to flow from the additional routine to the relocated opcodes; 

third means, coupled to first means and to the API, for causing program control to 
flow from the relocated opcodes to a memory address immediately following the 
relocatable portion; and 

fifth means, coupled to the API and to the additional routine, for causing program 
control to flow from the relocatable portion to the additional routine. 
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